Generally, a multispectral band of a remote sensor covers a narrower spectral range than a panchromatic band. The size of the sensor's detectors for the multispectral bands is larger than that for the panchromatic band to receive the same amount of incoming energy. Therefore, most earth observation satellites, such as the SPOT, IRS, Landsat 7, IKONOS and QuickBird, provide simultaneously low-resolution multispectral and high-resolution panchromatic images.
Many remote sensing applications require images with both high spectral resolution and high spatial resolution. The multispectral images provide high spectral resolution, but low spatial resolution. On the other hand, the panchromatic images provide high spatial resolution, but low spectral resolution. Thus, methods for effectively fusing (combining) the multispectral and panchromatic images to produce fused high spatial resolution (also called pan-sharpened) colour images are important. As a result, many image fusion methods have been developed, such as IHS (Intensity, Hue, Saturation), PCA (Principal Components Analysis), wavelet based fusion and SVR (Synthetic Variable Ratio). Among existing methods, the IHS and PCA fusion approaches have been the most popular ones.
Pohl and Van Genderen (1998) provide a comprehensive review of most conventional fusion techniques (Multisensor image fusion in remote sensing: concepts, methods and applications. International Journal of Remote Sensing, 1998, Vol. 19, No. 5, pp. 823-854).
However, the available fusion techniques contain two major problems: (1) the colour distortion of the fused image is significant, especially for the new satellite images of Landsat 7, IKONOS and QuickBird; and (2) the fusion quality is operator dependent. An optimal fusion quality is defined as having (1) minimized colour distortion (almost identical to the colour of original image), (2) maximized spatial detail (containing all the detail of the panchromatic image), and (3) natural (smooth) integration of colour and spatial feature from multispectral and panchromatic images. Unfortunately, available fusion techniques cannot produce such optimal fusion quality without intensive manual intervention. Moreover, even with intensive manual intervention, the available fusion techniques can still not fuse well the multispectral and panchromatic images from the new satellites Landsat 7, IKONOS, and QuickBird.
RE (Ratio Enhancement) Method
The RE method is a fusion technique which has some similarity to the present invention. The RE technique is described in Munechika et al. 1993 (Resolution Enhancement of Multispectral Image Data to Improve Classification Accuracy. Photogrammetric Engineering and Remote Sensing, Vol. 59, No. 1, pp. 67-72), and in section “4.3.1.2. Difference and ratio images” of Pohl and Van Genderen (1998) (see above). The equations used in the RE fusion are similar to equations (1) and (2) below.
However, in Munechika et. al (1993) Landsat TM and SPOT pan images were fused, and the weighting factors φi were calculated according to the simulated digital signals of Landsat TM bands 1, 2, 3 and 4. These signals were simulated using a modified version of LOWTRAN equations and representative pixels manually selected from five land cover classes in the image area. Multiple regression was involved in the calculation of the weightings φi.
Disadvantages and problems of the RE technique are that the method for obtaining φi is (i) operator dependent (manual selection of representative pixels of different classes), (ii) complicated (atmospheric models, LOWTRAN, are required to determine the digital signals of the selected pixels), and (iii) time consuming (interactive processes involving manual work and digital calculations are required), such that it is not suitable for industrial applications. Because digital signals of representative pixels from the five land cover classes were calculated to simulate the multispectral bands (the TM bands), the φi obtained was not a good representation of the colour characteristics of whole multispectral image. Further, the total spectral wavelength coverage of the TM bands 1, 2, 3 and 4 (0.45 to 0.90 μm) exceeds significantly the spectral coverage of the SPOT pan image (0.51 to 0.73 μm). Consequently, the simulated PanSyn image (see equation (2) below) did not represent well the grey value characteristics of the original multispectral and panchromatic images.
The fusion result was thus not optimal. This is a fatal problem of this technique. Even the authors themselves recognized this problem and tried to further modify the method for better results. However, further modifications, illustrated in the same paper, did not show any improvement in quality.
The Zhang SVR (Synthetic Variable Ratio) Method
The Zhang SVR method has some common components to the present invention. It was described in Zhang, Y., 1998: Vergleich verschiedener Verfahren zur Kombination multispectraler Satelliten-Bilddaten, Photogrammetrie-Fernerkundung-Geoinformation, May 1998, pp. 261-274; and Zhang, Y., 1999: A new merging method and its spectral and spatial effects. International Journal of Remote Sensing, Vol. 20, No. 10, pp. 2003-2014 which are incorporated herein by reference. The fusion principle can be described using equations (1), (2) and (3) below:
                              Fused          ⁢                                          ⁢                      Band            i                          =                              Multi            i                    ×                                    Pan              Orig                                      Pan              Syn                                                          (        1        )            with i=1, 2, 3 for three multispectral bands, and
                              Pan          Syn                =                  ∑                                          ⁢                                    φ              i                        ⁢                          Multi              i                                                          (        2        )            where Multii are the three multispectral bands to be pan-sharpened, and PanOrig is the original panchromatic band. The weights φi for calculating the synthetic panchromatic band PanSyn are solved using multiple regression directly from the three multispectral bands being fused:
                              Pan          Orig                =                  ∑                                          ⁢                                    φ              i                        ⁢                          Multi              i                                                          (        3        )            
The fusion process of the Zhang SVR method is described in FIG. 1. For example, if the multispectral bands TM 1, 2, 3 are being pan-sharpened, the same bands are used in equation (3) for calculating the φi (step 1 in FIG. 1), and then the same bands are used in equation (2) for calculating the PanSyn (step 2 in FIG. 1). The pan-sharpened TM 1, 2, and 3 bands are generated using equation (1) by multiplying individual multispectral bands with the ratio of original pan to synthetic pan (step 3 in FIG. 1). If TM 4, 5, 7 are being fused, TM 4, 5, 7 will be used for calculating the φi and then the PanSyn.
These changes were the major improvements over the RE method suggested by Munechika et al. (1993). These improvements eliminated the manual selection of representative pixels from classes of interest so that the process of calculating φi became operator independent. Only the multispectral bands being pan-sharpened are used in the fusion process so that the fusion process is simplified. The φi represent the colour characteristics of the whole image, instead of only the classes of interest, so that colour distortion is reduced. As a result, the quality of the fused images is better in terms of colour fidelity and spatial detail.
However, the total spectral coverage of the multispectral bands used for simulating the synthetic pan image varies significantly from that of the original panchromatic image from case to case. Thus, the fusion quality of the SVR method is heavily dependant upon the operator's fusion experience when SPOT pan or IRS pan is fused with low resolution multispectral images. Moreover, satisfactory fusion results cannot be achieved when images from the “new” satellites such as IKONOS, QuickBird, and Landsat 7 are fused. All the IKONOS, QuickBird or Landsat 7 fusions contain significant colour distortion.
Limitation of Other Conventional Fusion Methods
Similar to the SVR method, other conventional fusion methods, such as IHS (Intensity, Hue, Saturation), PCA (Principal Components Analysis) and wavelet fusion, are also operator dependent. Different operators, source data from different areas or data from different times may result in different fusion results.
Moreover, none of them produces good results when used to fuse the multispectral and panchromatic images of “new” satellites such as IKONOS, QuickBird, and Landsat 7. The colour of the fused images is significantly distorted when compared with the original colour of the multispectral images. Such colour distortion has been demonstrated by many image fusion experts, even though manual intervention and colour adjustment were employed. It has been found that the colour distortion varies from scene to scene, and the colour of many pan-sharpened images does not look natural, especially in vegetation areas.
There is, therefore, a need for a new image fusion method that eliminates the need for operator intervention for colour adjustments and, more important, produces fused images of a consistently satisfactory quality.